{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "37v_yExZppEp"
   },
   "source": [
    "# Ungraded Lab: Data Augmentation on the Horses or Humans Dataset\n",
    "\n",
    "In the previous lab, you saw how data augmentation helped improve the model's performance on unseen data. By tweaking the cat and dog training images, the model was able to learn features that are also representative of the validation data. However, applying data augmentation requires good understanding of your dataset. Simply transforming it randomly will not always yield good results.\n",
    "\n",
    "In the next cells, you will apply the same techniques to the `Horses or Humans` dataset and analyze the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5M2UgUGbOAE7",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# First import the necessary libraries\n",
    "import os\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RXZT2UsyIVe_"
   },
   "outputs": [],
   "source": [
    "TRAIN_DIR = 'horse-or-human/train'\n",
    "VAL_DIR = 'horse-or-human/validation'\n",
    "\n",
    "# Directory with training horse pictures\n",
    "train_horse_dir = os.path.join(TRAIN_DIR, 'horses')\n",
    "\n",
    "# Directory with training human pictures\n",
    "train_human_dir = os.path.join(TRAIN_DIR, 'humans')\n",
    "\n",
    "# Directory with validation horse pictures\n",
    "validation_horse_dir = os.path.join(VAL_DIR, 'horses')\n",
    "\n",
    "# Directory with validation human pictures\n",
    "validation_human_dir = os.path.join(VAL_DIR, 'humans')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PixZ2s5QbYQ3"
   },
   "outputs": [],
   "source": [
    "def create_model():\n",
    "  '''Builds a CNN for image binary classification'''\n",
    "\n",
    "  model = tf.keras.models.Sequential([\n",
    "      tf.keras.Input(shape=(300,300,3)),\n",
    "      # This will rescale the image to [0,1]\n",
    "      tf.keras.layers.Rescaling(1./255),\n",
    "      # This is the first convolution\n",
    "      tf.keras.layers.Conv2D(16, (3,3), activation='relu'),\n",
    "      tf.keras.layers.MaxPooling2D(2, 2),\n",
    "      # The second convolution\n",
    "      tf.keras.layers.Conv2D(32, (3,3), activation='relu'),\n",
    "      tf.keras.layers.MaxPooling2D(2,2),\n",
    "      # The third convolution\n",
    "      tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
    "      tf.keras.layers.MaxPooling2D(2,2),\n",
    "      # The fourth convolution\n",
    "      tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
    "      tf.keras.layers.MaxPooling2D(2,2),\n",
    "      # The fifth convolution\n",
    "      tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
    "      tf.keras.layers.MaxPooling2D(2,2),\n",
    "      # Flatten the results to feed into a DNN\n",
    "      tf.keras.layers.Flatten(),\n",
    "      # 512 neuron hidden layer\n",
    "      tf.keras.layers.Dense(512, activation='relu'),\n",
    "      # Only 1 output neuron. It will contain a value from 0-1 where 0 for one class ('horses') and 1 for the other ('humans')\n",
    "      tf.keras.layers.Dense(1, activation='sigmoid')\n",
    "      ])\n",
    "\n",
    "  return model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oAhbnBwQeQWi"
   },
   "outputs": [],
   "source": [
    "# Constants\n",
    "BATCH_SIZE = 32\n",
    "IMAGE_SIZE = (300, 300)\n",
    "LABEL_MODE = 'binary'\n",
    "\n",
    "# Instantiate the training dataset\n",
    "train_dataset = tf.keras.utils.image_dataset_from_directory(\n",
    "    TRAIN_DIR,\n",
    "    image_size=IMAGE_SIZE,\n",
    "    batch_size=BATCH_SIZE,\n",
    "    label_mode=LABEL_MODE\n",
    "    )\n",
    "\n",
    "# Instantiate the validation set\n",
    "validation_dataset = tf.keras.utils.image_dataset_from_directory(\n",
    "    VAL_DIR,\n",
    "    image_size=IMAGE_SIZE,\n",
    "    batch_size=BATCH_SIZE,\n",
    "    label_mode=LABEL_MODE\n",
    "    )\n",
    "\n",
    "# Optimize the datasets for training\n",
    "SHUFFLE_BUFFER_SIZE = 1000\n",
    "PREFETCH_BUFFER_SIZE = tf.data.AUTOTUNE\n",
    "\n",
    "train_dataset_final = (train_dataset\n",
    "                       .cache()\n",
    "                       .shuffle(SHUFFLE_BUFFER_SIZE)\n",
    "                       .prefetch(PREFETCH_BUFFER_SIZE)\n",
    "                       )\n",
    "\n",
    "validation_dataset_final = (validation_dataset\n",
    "                            .cache()\n",
    "                            .prefetch(PREFETCH_BUFFER_SIZE)\n",
    "                            )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BvaHC3PhAwpa"
   },
   "source": [
    "## Data Augmentation\n",
    "\n",
    "You will use the same layers as in the previous lab. This worked well for the Cats and Dogs dataset to avoid overfitting. See if it has the same effect here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ta9VEsUkfRjQ"
   },
   "outputs": [],
   "source": [
    "# Define fill mode.\n",
    "FILL_MODE = 'nearest'\n",
    "\n",
    "# Create the augmentation model.\n",
    "data_augmentation = tf.keras.Sequential([\n",
    "        # Specify the input shape.\n",
    "        tf.keras.Input(shape=(300,300,3)),\n",
    "        # Add the augmentation layers\n",
    "        tf.keras.layers.RandomFlip(\"horizontal\"),\n",
    "        tf.keras.layers.RandomRotation(0.2, fill_mode=FILL_MODE),\n",
    "        tf.keras.layers.RandomTranslation(0.2, 0.2, fill_mode=FILL_MODE),\n",
    "        tf.keras.layers.RandomZoom(0.2, fill_mode=FILL_MODE)\n",
    "        ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Iu6K7lHKDCgN"
   },
   "source": [
    "You will again use the `data_augmentation` model on some images to preview the transformations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GDsuI9YpUT_J"
   },
   "outputs": [],
   "source": [
    "def demo_augmentation(sample_image, model, num_aug):\n",
    "    '''Takes a single image array, then uses a model to generate num_aug transformations'''\n",
    "\n",
    "    # Instantiate preview list\n",
    "    image_preview = []\n",
    "\n",
    "    # Convert input image to a PIL image instance\n",
    "    sample_image_pil = tf.keras.utils.array_to_img(sample_image)\n",
    "\n",
    "    # Append the result to the list\n",
    "    image_preview.append(sample_image_pil)\n",
    "\n",
    "    # Apply the image augmentation and append the results to the list\n",
    "    for i in range(NUM_AUG):\n",
    "        sample_image_aug = model(tf.expand_dims(sample_image, axis=0))\n",
    "        sample_image_aug_pil = tf.keras.utils.array_to_img(tf.squeeze(sample_image_aug))\n",
    "        image_preview.append(sample_image_aug_pil)\n",
    "\n",
    "    # Instantiate a subplot\n",
    "    fig, axes = plt.subplots(1, NUM_AUG + 1, figsize=(12, 12))\n",
    "\n",
    "    # Preview the images.\n",
    "    for index, ax in enumerate(axes):\n",
    "        ax.imshow(image_preview[index])\n",
    "        ax.set_axis_off()\n",
    "\n",
    "        if index == 0:\n",
    "            ax.set_title('original')\n",
    "        else:\n",
    "            ax.set_title(f'augment {index}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JBUQLcz-Cg6F"
   },
   "outputs": [],
   "source": [
    "NUM_AUG = 4\n",
    "\n",
    "# Get a batch of images\n",
    "sample_batch = list(train_dataset.take(1))[0][0]\n",
    "\n",
    "# Apply the transformations to the first 4 images\n",
    "demo_augmentation(sample_batch[0], data_augmentation, NUM_AUG)\n",
    "demo_augmentation(sample_batch[1], data_augmentation, NUM_AUG)\n",
    "demo_augmentation(sample_batch[2], data_augmentation, NUM_AUG)\n",
    "demo_augmentation(sample_batch[3], data_augmentation, NUM_AUG)\n",
    "\n",
    "# Delete the variable to free up some memory\n",
    "del sample_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A-pVur9vEf0o"
   },
   "source": [
    "Now create a model attaching the `data_augmentation` model to the base model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8DHWhFP_uhq3"
   },
   "outputs": [],
   "source": [
    "# Instantiate the base model\n",
    "model_without_aug = create_model()\n",
    "\n",
    "# Prepend the data augmentation layers to the base model\n",
    "model_with_aug = tf.keras.models.Sequential([\n",
    "    data_augmentation,\n",
    "    model_without_aug\n",
    "])\n",
    "\n",
    "# Compile the model\n",
    "model_with_aug.compile(loss='binary_crossentropy',\n",
    "              optimizer=tf.keras.optimizers.RMSprop(learning_rate=1e-4),\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9LC1X4pwErfa"
   },
   "source": [
    "Now train the model for 20 epochs and observe what's happening with the training and validation accuracy. You will plot the results to see it more clearly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Fb1_lgobv81m"
   },
   "outputs": [],
   "source": [
    "# Constant for epochs\n",
    "EPOCHS = 20\n",
    "\n",
    "# Train the model\n",
    "history = model_with_aug.fit(\n",
    "      train_dataset_final,\n",
    "      epochs=EPOCHS,\n",
    "      verbose=2,\n",
    "      validation_data = validation_dataset_final)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7zNPRWOVJdOH"
   },
   "outputs": [],
   "source": [
    "def plot_loss_acc(history):\n",
    "    '''Plots the training and validation loss and accuracy from a history object'''\n",
    "    acc = history.history['accuracy']\n",
    "    val_acc = history.history['val_accuracy']\n",
    "    loss = history.history['loss']\n",
    "    val_loss = history.history['val_loss']\n",
    "\n",
    "    epochs = range(len(acc))\n",
    "\n",
    "    fig, ax = plt.subplots(1,2, figsize=(12, 6))\n",
    "    ax[0].plot(epochs, acc, 'bo', label='Training accuracy')\n",
    "    ax[0].plot(epochs, val_acc, 'b', label='Validation accuracy')\n",
    "    ax[0].set_title('Training and validation accuracy')\n",
    "    ax[0].set_xlabel('epochs')\n",
    "    ax[0].set_ylabel('accuracy')\n",
    "    ax[0].legend()\n",
    "\n",
    "    ax[1].plot(epochs, loss, 'bo', label='Training Loss')\n",
    "    ax[1].plot(epochs, val_loss, 'b', label='Validation Loss')\n",
    "    ax[1].set_title('Training and validation loss')\n",
    "    ax[1].set_xlabel('epochs')\n",
    "    ax[1].set_ylabel('loss')\n",
    "    ax[1].legend()\n",
    "\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "IUln7ESeVdTn"
   },
   "outputs": [],
   "source": [
    "# Plot training results\n",
    "plot_loss_acc(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hwyabYvCsvtn"
   },
   "source": [
    "As you can see in the results, the preprocessing techniques used in augmenting the data did not help much in the results. The validation accuracy is fluctuating and not trending up like the training accuracy. This might be because the additional training data still do not represent the features in the validation data. For example, some human or horse poses in the validation set cannot be mimicked by the augmented data. It might also be that the background of the training images are also being learned by the model so the white background of the validation set is throwing the prediction off. (_Optional: you can learn about mismatched train/test distributions in [the later parts of this video](https://www.coursera.org/lecture/deep-neural-network/train-dev-test-sets-cxG1s)_).\n",
    "\n",
    "Use the function below to compare the images on both datasets (note: you can also use the file explorer on the left to explore the images)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "uXuzCplUXDzn"
   },
   "outputs": [],
   "source": [
    "def show_images(dir_1, dir_2):\n",
    "    '''plots 8 images each from two given directories'''\n",
    "\n",
    "    # Output images in a 2x8 configuration\n",
    "    nrows = 2\n",
    "    ncols = 8\n",
    "\n",
    "    # Get filenames from the two directories\n",
    "    fnames_dir_1 = os.listdir(dir_1)\n",
    "    fnames_dir_2 = os.listdir(dir_2)\n",
    "\n",
    "    # Set up matplotlib fig, and size it to fit 2x8 images\n",
    "    fig = plt.gcf()\n",
    "    fig.set_size_inches(ncols * 4, nrows * 4)\n",
    "\n",
    "    # Get 8 random filenames and set the full paths\n",
    "    next_pix_dir_1 = [os.path.join(dir_1, fname)\n",
    "                      for fname in random.sample(fnames_dir_1, 8)]\n",
    "    next_pix_dir_2 = [os.path.join(dir_2, fname)\n",
    "                      for fname in random.sample(fnames_dir_2, 8)]\n",
    "\n",
    "    # Plot the chosen images\n",
    "    for i, img_path in enumerate(next_pix_dir_1 + next_pix_dir_2):\n",
    "\n",
    "        # Set up subplot. subplot indices start at 1\n",
    "        sp = plt.subplot(nrows, ncols, i + 1)\n",
    "\n",
    "        # Don't show axes or gridlines\n",
    "        sp.axis('Off')\n",
    "\n",
    "        # Load and show the image\n",
    "        img = tf.keras.utils.load_img(img_path)\n",
    "        plt.imshow(img)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FKPMLaZUJTBg"
   },
   "outputs": [],
   "source": [
    "# Show random train (first row) and validation (second row) horse images\n",
    "show_images(train_horse_dir, validation_horse_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "egcXuissdiaH"
   },
   "outputs": [],
   "source": [
    "# Show random train (first row) and validation (second row) human images\n",
    "show_images(train_human_dir, validation_human_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "htv0hu2DaoVz"
   },
   "source": [
    "Based on these and the preview of the tranformed images earlier, see if it's possible to modify the data augmentation model to avoid overfitting. Getting satisfactory results can be difficult, and at this point, you can consider other techniques. You will see that in next week's lessons."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "private_outputs": true,
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0rc1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
